COTS Software Selection: The Need to make Tradeoffs between System Requirements, Architectures and COTS/Components

This short paper presents a new research agenda to address problems of COTS software selection in the forthcoming decade. It describes the increasing shift towards software engineering based on COTS software packages, the limitations of current COTS/component-based software engineering methods and research efforts, and proposes a new research agenda to address the problems which arise from a software engineering process based on the reuse of COTS software and software components. The focus of the proposed research is how can we develop complex software systems by integrating together different combinations of COTS software packages and software components. 1. The Changing Face of Software Engineering There are few methods, guidelines or environments that support the selection of COTS and component software. This is an increasing problem. Forrester Research estimate that 70% of European software development will be componentor COTSbased by 2003. The coverage of the problem is also large. In business systems engineering, enterprise integration applications will need to be extended with business software components. In defence systems the paradigm is shifting towards more software and component procurement from third parties. In both business and defence systems engineering, the selection of COTS software and software components which meet customer requirements and, at the same time, fit the system architecture and do not give rise to undesirable emergent behaviour is THE problem. The solution is often a trade-off between the satisfaction of stakeholder requirements and architecture requirements. The need to be able to make such trade-offs, and the theoretical and applied research needed to achieve these trade-offs, are the foci of this paper. The shift towards software procurement and outsourcing means that prime contractors for software systems will need to have significant new capabilities to undertake tasks previously undertaken by customers. These tasks include evaluating and managing total system risks, analysing trade-offs between system performance, delivery deadlines and whole life cycle costs, and facilitating effective communication with stakeholders. However, prime contractors lack effective tools and techniques for these crucial tasks. For many prime contractors the software development process is too solution-driven and compartmentalised around software components and COTS software packages to be applied to a system which is itself composed of complex inter-operating sub-systems. A step increase is required in the capabilities to model all of the perspectives of a complex system and its production to handle increasing system complexity, value-based trade-offs, conformance assessment of requirements, and risk and uncertainty management. Prime contractors need new processes for developing complex systems with COTS software and software components, capabilities to detect the weak points in the processes and how those processes might be improved. It also requires new integrated system capabilities for determining system requirements, for evaluating and implementing design trade-offs, and for handling risks and uncertainties associated with such trade-offs using COTS software components. The selection, evaluation and integration of COTS software and software components is critical in making trade-off decisions to optimise the final software system, however most COTS/component-based software engineering research does not recognise this. The next section describes a current method for COTS software selection and discusses its weaknesses with respect to the need to deliver complex systems composed of many different COTS software and software components. The remainder of the paper describes part of a research agenda to deliver the capabilities needed by prime contractors. 2. Current Methods for COTS Software Selection There are few methods currently available to support the selection and integration of COTS software and software components that meet customer requirements. One exception that we will outline here is PORE. The PORE (Procurement-Oriented Requirements Engineering) method guides a software development team to acquire customer requirements and select COTS software systems which satisfy these requirements (Ncube 1999). It supports an iterative process of requirements acquisition and COTS software selection. During each iteration, the software development team: (1) acquires customer requirements which discriminate between the COTS software candidates; (2) undertakes multi-criteria decision-making to identify candidates that are not compliant with these requirements; (3) rejects COTS software candidates that are non-compliant with the requirements; (4) explores the remaining COTS software candidates to discover possible new customer requirements which might discriminate further between these candidates. The customer requirements to acquire in each iteration change throughout the process as the decisions that the team has to make change. The team continues these iterations until, in an ideal situation, one or more candidate COTS software which comply with all essential customer requirements remain. Figure 1 depicts this iterative process. acquire customer requirements check compliance with COTS software reject noncompliant COTS software explore remaining COTS software candidates Figure 1. PORE’s iterative process of requirements acquisition and COTS software selection. PORE offers techniques to acquire customer requirements that enable the evaluation team to discriminate between COTS software candidates. PORE also offers techniques such as scenarios to discover, acquire and structure customer requirements to formulate test cases that are used to check COTS software compliance with the customer requirements. Other PORE techniques enable the team to discover and select cornerstone COTS software packages before peripheral packages which integrate with it. PORE also recommends that stakeholder representatives are present during COTS software demonstrations to ask important but unforeseen questions and to discover missing customer requirements. The complete description of the techniques offered in PORE is reported in Maiden & Ncube (1998). Another feature of PORE is that it guides the software development team to make the decisions to select or reject COTS software candidates. Each iteration of the PORE process guides the team to acquire different customer requirements to reject COTS software that are not compliant with these requirements. The sequence of the decisions that PORE guides the team to make is shown in Figure 2. More simple decisions (e.g. to reject COTS software candidates that are not compliant with simple requirements such as cost) are made earlier in the process when there are more candidates. More complex decisions (e.g. to reject software candidates for noncompliance with complex customer requirements such as system performance) are made later in the process when the COTS software is available to explore complex emergent system properties. Figure 2: the sequence of critical decision-making stages to reject COTS software in PORE. The PORE method has been evaluated on a number of case studies to be reported in Ncube & Maiden (2000), and has been shown to be useful and usable to customer organisations. However, the current limitations of PORE are clear. It only supports the selection of one COTS software package at a time, does not support integration of the COTS software and components to explore emergent system properties essential to test compliance with system requirements, and as a result does not scale to the types of systems which are commonplace in today’s business and defence systems. 3. A Research Agenda for COTS-Based Systems Integration Organisations need new methods, processes and modelling environments to undertake trade-off analyses that inform COTS and component selection. The envisaged research solution is large. The remainder of this paper will investigate some core intellectual problems which, when solved, can enable a wider solution. It will propose a new version of the PORE method extended with a modelling environment that will support trade-off analyses to enable COTS/component selection. The new PORE method will have 2 extensions. First, it will have a modelling environment which will act as a ‘test rig’ to explore the impact of different COTS/component software on stakeholder and architectural requirements for a time t Acquired requirements Template-2 Template-3 Template-4 compliance m appings Candidate COTS-components under consideration Template-1 Select using simple software component and supplier information Select using information from supplier demonstrations Select using information from hands-on COTS component use Select using emergent system properties from user trials system. Figure 3 depicts this ‘test rig’ environment. The vision is a “plug-and-play” environment in which different COTS/component combinations can be explored quickly to assess how system performance varies with these different combinations. The environment will use models of COTS/component software (possibly derived with reverse engineering techniques) rather than COTS software itself (Voas 1998). This has implication for libraries of COTS/component software. Figure 4 depicts this ‘plug-and-play’ environment. Second, the method will be extended with multi-criteria decision making techniques to enable software engineers to make trade-offs between the system performance and the cost and time to deliver this COTS/component combination. These trade-offs are a central component of systems engineering and systems integration. Let us explore these extensions in more detail. Figure 3: A modelling and simulation environment is needed in order to understand many competing, critical issues that arise during requirements engineering for COTS-based development. This environment will take as input customer requirements (both functional and non-functional), current legacy systems, candidate software products, and the glue ware software to explore the selection and integration of COTS products into their application environment. One critical success factor will be a plug-and-play environment as depicted in Figure 4 below. The first extension is to categorise and model recurring problems which arise during COTS/component-based systems development as a basis for problem diagnosis in systems integration as depicted in Figure 4. There are 2 reasons for this. First, systems integration from COTS and component software is still a new area of research, and categorisation of systems integration problems and consequences will improve our understanding of detailed problems and hence drive an agenda for more specific Requirements